Skip to content

[SPARK-10169] [SQL] [BRANCH-1.4] Partial aggregation's plan is wrong when a grouping expression is used as an argument of the aggregate fucntion#8379

Closed
yhuai wants to merge 3 commits intoapache:branch-1.4from
yhuai:aggTransformDown-branch1.4
Closed

[SPARK-10169] [SQL] [BRANCH-1.4] Partial aggregation's plan is wrong when a grouping expression is used as an argument of the aggregate fucntion#8379
yhuai wants to merge 3 commits intoapache:branch-1.4from
yhuai:aggTransformDown-branch1.4

Conversation

@yhuai
Copy link
Contributor

@yhuai yhuai commented Aug 23, 2015

@SparkQA
Copy link

SparkQA commented Aug 23, 2015

Test build #41418 has finished for PR 8379 at commit 009aadb.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yhuai
Copy link
Contributor Author

yhuai commented Aug 23, 2015

hmm. seems it triggers another bug?

btw, the failed test is org.apache.spark.sql.SQLQuerySuite.grouping on nested fields.

@cloud-fan
Copy link
Contributor

Before 1.5, ExtractValue chain may have intermedia and unnecessary Aliases inside it, so we trim Aliases upon ExtractValue before compare 2 Expressions in a lot places including here.
However, when we use transformDown, we may loose the top level Alias. For example, 'a.getField("b") is a grouping expression and 'a.getField("b").as("attribute") is an aggregate expression,
with transformDown, the rewritten aggregate expression will be 'a.getField("b") which is wrong.

The commit to fix the wrong Alias trimming is da7bbb9, which is not easy to backport I think. So maybe we need to add some ugly code to hadle the top level Aliases specially.

@cloud-fan
Copy link
Contributor

btw I have sent a PR on your branch, let's see if that can work :)

@SparkQA
Copy link

SparkQA commented Aug 23, 2015

Test build #41426 has finished for PR 8379 at commit afcfad9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

LGTM

asfgit pushed a commit that referenced this pull request Aug 24, 2015
…when a grouping expression is used as an argument of the aggregate fucntion

https://issues.apache.org/jira/browse/SPARK-10169

Author: Yin Huai <yhuai@databricks.com>
Author: Wenchen Fan <cloud0fan@outlook.com>

Closes #8379 from yhuai/aggTransformDown-branch1.4.
@marmbrus
Copy link
Contributor

Thanks guys, merged!

@yhuai yhuai closed this Aug 24, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants